An Environment for Conducting Families of Software Engineering Experiments

The classroom is a valuable resource for conducting software engineering experiments. However, coordinating a family of experiments in classroom environments presents a number of challenges to researchers. Understanding how to run such experiments, developing procedures to collect accurate data, and collecting data that is consistent across multiple studies are major problems. ADVANCES IN COMPUTERS, VOL. 74 175 Copyright © 2008 Elsevier Inc. ISSN: 0065-2458/DOI: 10.1016/S0065-2458(08)00605-0 All rights reserved. 176 L. HOCHSTEIN ET AL. This paper describes an environment, the Experiment Manager that simplifies the process of collecting, managing, and sanitizing data from classroom experiments, while minimizing disruption to natural subject behavior. We have successfully used this environment to study the impact of parallel programming languages in the high-performance computing domain on programmer productivity at multiple universities across the United States.